The present invention is related generally to software and computer programs. More specifically, the present invention is related to software for sizing and specifying database management system hardware.
Businesses and other organizations implementing computer systems, and specifically database management systems (DBMS), and relational database management systems (RDBMS), have naturally been interested in obtaining some measure of the performance of systems they are considering, to enable comparison shopping among competing systems. This interest extends both to the hardware systems available for their database system, for example, its speed and capacity, and to the commercial software systems available to run on the hardware systems. The desire to have some objective measure of the performance of a system, and how this system performs relative to competing systems, is natural in view of the competing claims made by different hardware and software vendors. Not only is the conflicting xe2x80x9cpuffingxe2x80x9d of sales representatives not helpful to the purchasing decision, but even seemingly objective measures of a system""s capabilities may be influenced by the tests that a vendor uses to demonstrate their system. In other words, vendors will tend to use demonstration or evaluation criteria that emphasize their product""s strong points, and downplay or minimize areas in which their system is weaker than their competitors"".
Several benchmark standards have been proposed, in order to provide a relatively level playing field with respect to the evaluation of different systems. Typically, benchmarking of database systems involves the construction of a hypothetical or example database. Predefined functions such as queries and/or updates, typically specified in the benchmark in SQL, are executed on this database using the hardware or software system being considered, and the database system must provide accurate results. If the database system gives proper output to the queries submitted, and updates the database accurately, the speed, throughput, or efficiency of the system may be analyzed. Two early benchmarks for DBMS, introduced in the 1980s, were the Wisconsin benchmark and the Debit-Credit benchmark. Subsequently, a benchmarking system named TP1 was created, which tested the ability of a DBMS to handle database functions related to cashing a check. One shortcoming of some of these early, relatively simple benchmarking systems was that it was possible for an unscrupulous vendor to xe2x80x98cheatxe2x80x99 by making insignificant changes to a hardware or software product that, while not improving the system as a whole for most real-world applications, would bring greatly improved performance on the benchmark test.
Criticisms of these early systems led to a multivendor effort to create a benchmarking system that would fairly test systems, and on which would not be possible to cheat. The multivendor efforts to create a fair benchmark led to the formation of the Transaction Processing Council, or TPC. One way that the TPC reduced a vendor""s opportunity to tune its system to perform well on a specific benchmark test was to provide for random variation in certain data to be inserted into the database or in queries issued to the system. Early standards were named TPC-A, TPC-B, and TPC-C. The council has published several xe2x80x98transaction processingxe2x80x99 and xe2x80x98decision supportxe2x80x99 benchmarking standards. Transaction processing benchmark specification analyze the ability of a given system to handle transactions related to, for example, individual customer""s accounts, or other OLTP (on-line transaction processing) functions. Decision support benchmarking tests a system""s ability to rapidly analyze entire stores of data, and return averages or transactions that have a parameter within a certain range of values. For example, a decision support benchmark database query may ask what the impact on revenue would be if sales of a certain range received a percentage discount.
More recent standards propagated by the TPC include TPC-D, TPC-H and TPC-R. The TPC results for various vendors"" systems are made publicly available at the Transaction Processing Council""s web site. The TPC-C benchmark, for example, has two chief parameters. The tpmC (xe2x80x9ctransactions per minute (C)xe2x80x9d) and $/tpmC (xe2x80x9cprice per transactions per minute (C)xe2x80x9d). The tpmC metric provides a rough measure of xe2x80x9cbusiness throughput,xe2x80x9d representing the number of orders processed on a database system per minute. The $/tpmC represents the cost of the system for each transaction per minute. $/tpmC is derived by dividing the price of the entire system, not merely the server, by the tpmC that the system delivered in the benchmark evaluation. $/tpmC provides a measure of the xe2x80x9cbang for the buck,xe2x80x9d or in other words, the cost of the system adjusted for differences in speed between systems.
To simulate the business activity of processing an order, the following transactions are simulated under the TPC-C benchmark: New-Order, Payment, Order-Status, Delivery, and Stock-Level. Transactions per minute (tpmC) measures the number of New Order transactions that may be processed per minute by the computer being considered.
While the TPC benchmark results provide a valuable resource for consideration of the performance of various systems, an extensive number of different systems, and different hardware and software combinations, are available and included on the TPC site. These results are voluminous, and not readily scanned by human beings to make accurate or good overall judgments about what computer system will deliver the best performance for a given price range, or deliver a desired level of performance for the best price. It is desirable, therefore, to provide a convenient environment for rapid consideration of benchmark performance across systems in order to estimate the relative performance and value of various computer systems that a system planner may be considering.
The instant invention provides an environment for the consideration of various hardware and software combinations for a DBMS, and provides for the accurate quantitative comparison of competing systems according to their benchmark performance.
In a preferred embodiment, the statistical data used in the comparison method is that published by the Transaction Processing Council. Other statistical compilations of server performance may also be used, including those by other organizations, other TPC performance criteria, proprietary statistics, statistics furnished by vendors in promoting certain equipment, etc.
In one illustrative embodiment of the present invention, a system planner is presented with an option to select a configuration for a baseline system. This is the system against which a target system""s performance will be compared. The system planner first selects from a choice of operating systems. These may include common network operating systems such as Unix, Windows NT, Novell NetWare, IBM AS/xxx, etc. In the event that the system planner wishes to consider operating systems that are not specifically presented, an option may be presented for a general analysis leaving the operating system unspecified.
The system planner may also select a Database Management System (or DBMS) that is or may be used with the platform configuration selected. The system planner is preferably presented with common database software such as DB2, Informix, Oracle, SQL Server, Sybase, etc. In the event the system planner wishes to consider database software that is not specifically presented, an option may be provided for a general analysis leaving the database software unspecified. This scenario may occur, for example, when proprietary or xe2x80x98in-housexe2x80x99 database software is used, where performance data is not likely to be available.
After selection of a baseline system, the system planner is presented with the configuration options for a xe2x80x98targetxe2x80x99 system. The system planner then selects the parameters of the system that will be objectively compared with the initial xe2x80x98baselinexe2x80x99 system. In one embodiment, the invention allows the system planner to easily determine which system is more cost-effective for a desired application, without having to adjust performance statistics for the cost of the computer systems involved.
It is contemplated that the present invention may prevent the system planner from comparing the benchmark data for database servers of two different sizes, when the comparison is likely to be misleading or inaccurate. Benchmark organizations often caution against considering benchmark statistics between two systems with widely disparate memory, mass storage capacity, number of processors, or processor speed. In such scenarios, the present invention may issue warnings regarding the limitations of the benchmarking data in comparing two systems with widely varying characteristics.
Likewise, it is contemplated that the present invention may prevent the system planner from comparing benchmark data or other performance statistics when the proponents of the statistics do not consider the comparison to be statistically reliable. For example, the benchmarking organization may advise that certain benchmark data may be valid for only a limited period of time. Often, improvements in hardware products or software applications where no new version number is indicated may render benchmark test data unreliable. Alternatively, or in addition to, the benchmark test itself may be disfavored in view of improved benchmark tests. In these situations, the present invention may notify the system planner of limitations in the benchmark data.
An alternate embodiment may cause a benchmarking database to expire and become inaccessible a certain length of time after it is downloaded or loaded into a computer adapted to perform the invention. This may prevent the system planner from relying on untimely data when making system planning decisions.
Once the baseline system and target system are selected, the ratios of the baseline system""s performance parameter to the target system""s performance parameter, and/or the ratio of the baseline system""s price to performance ratio to the target system""s price to performance ratio, are displayed. Preferably, these ratios are dynamically displayed as soon as any aspects of the baseline or target system are changed, so that the system planner does not have to select criteria and then resubmit the system characteristics or enter a command to recalculate the ratios based on the new criteria.
In another illustrative embodiment, the TPC-D database may be considered. For example, instead of the transactions per minute metric of the TPC-C database, a performance variable called QppD@Size may be considered. Under this metric, the relative database size is used as a scale factor to determine the population of each table in the database. A scale factor (SF) of 1, for example, corresponds to about 1 GB of raw data.
QppD@Size=[(3600*SF)/(Q1*Q2* . . . *Q17)*UF1*UF2]1/19, where Qi=Elapsed time to run query i within a single query stream; UF1=Elapsed time to run update function 1; UF2=Elapsed time to run update function 2; SF=Scaling Factor. The divisor of the equation represents the geometric mean of the timing intervals, i.e., the elapsed time to run the queries of the test, plus two update functions. Because the elapsed time entries into the metric are in seconds, the complete power metric QppD@Size has units of queries per hour.
The scale factor is unitless in the equation. The 3600 figure, also without units, converts the function per second of the geometric mean of the timing intervals into the queries per hour units of the QppD@Size metric. The throughput metric of the TPC-D benchmark database, QthD@Size, is the ratio of the number of queries executed to the length of the measurement interval in seconds: QthD@Size=(N*17*3600*SF)/L; where N =the number of query streams, and L=the length of the measurement interval in seconds. The units of the QthD@Size metric are queries per hour, adjusted for the scale factor, SF, to account for differences in database size.
Generally, both of these TPC-D parameters represent the number of queries that a hardware configuration can handle per gigabyte hour. Both the QppD@Size and the QthD@Size parameters are combined with system cost to provide the cost for a given level of throughput, measured as the geometric mean of both throughput parameters: $/QphD@Size=C/(QppD@Size*QthD@Size)1/2; where C=cost of system in dollars.
In yet another embodiment of the present invention, the invention may be implemented as a platform selection computer program. Under this embodiment, the user or system planner may enter a software application to be used, for example, in accessing and updating a relational database. The system planner may then select a particular relational database management system software application, for example, an application sold under the brand Oracle(trademark). The subject invention, implemented in an illustrative embodiment as a computer program, may then solicit from the system planner the range of performance required for the computer system that the system planner is implementing. For example, the system planner could request a list of all benchmarked systems that perform at a certain minimum number of transactions per minute (tpmC) or above. In response, the invention may supply the system planner with a report of all computer hardware systems benchmarked in a particular database, and the operating systems used with the machines, that ran the Oracle(trademark) application at least at the performance minimum set by the system planner.
In another illustrative embodiment, the system planner may simply input a minimum performance parameter, such as a maximum price of cost per transaction per minute ($/tpmC) under the TPC-C benchmark (TPC-C). In response, the invention may return a list of all database systems that meet the system planner""s specified parameters. Other embodiments of the invention may allow the system planner to input an existing hardware system and selected required parameters. In response, the invention may identify those operating system and/or software application configurations that meet the required parameters.
In a typical embodiment, the system planner interfacing with the invention will be a human user considering the purchase of a database management system. However, other embodiments are contemplated, including an embodiment where the system planner is a computer system, or is software running on the same or another computer system.